Estimating the accuracy of dichotomous judgments.

A reliability study is assumed to be carried out with each of a number of observers making a dichotomous ]udgment concerning each of a sample of subjects. A nonparametric model is proposed for the errors underlying such judgments, and conditions are given under which Cochran's Q statistic is valid for testing the hypothesis of no systematic differences among the judgments of the different observers. Inferences concerning the probabilities of error are shown to be possible in terms of the intraclass correlation coefficient. & numerical example is given.